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(57) Abstract:. An ultrasound imadrig system superimposes sectional views 
created from volumetric ultrasound data and the location data for an inter- 
vention device, such as a catheter. The position of an intjerventidnal medical 
device may be shown, in one or more views, relative to organs and tissues 
within a body as the interventional device is moved. The interventional de- 
vice posiu'onal data is updated continuously and is superimposed on tissue 
images th^ niay be updated less -frequently, resulting in real-time dir near 
real-time images of die interventional device ielative to the tissues. The su- 
perimposed images permits medical personnel to perform pro<^ures such 
as angiograms with minimal or ho exposure of patients to x-rays and con- 
trasting dye. The look and feel of the familiar fluOTOSCopy-like imaging may 
be maintained, or a three dimensional real-time, or near-time projected im- 
age of the intervention medical device relative to an organ or tissue may be 
provided.; 
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ULTRASONIC IMAGER 

BACKGRdUNlb OF THE IN^ 


. Field of the Invention 

The present invention relates to Wtrasonic imaging. More particularly, the 
5 preseht invention relates to creatihg sectional views from volumenic ultrasouttd data 
• ' arid superimposing positional data for an inierventional device, such zs a. catheter, ' 

Background of the In vention 

Many medical procedures involve insefung a catheter, of other interventional 
• device, into a patientrfor a variety of purposes. One such procedtire. is cardiac 
iO ...catheterization for an angiogram. 

To observe the condition of blood vessels wthin the heart, radio-bpaqtie dye 
is injected into the bloodstream through the catheter and ah x«ray image is takea. 
Typically, this procedure involves inseniiig a catheter into in artery in the patient's 
groin area and guiding the catheter through the anerial system and the aorta to the 
15 heart. In order to position the catheterand inomtor its location within the patient^ one 
■ or mbre x-ray images may be taken prior to 'injecting dye for the angiogram. 

A catheter niay be du-ectly visible in ari x-ray image, however, a heart typically 
is not in order to determine the locatibri of the catheter relative to the heart, dye may 
be injected through the catheter into the blopdstream/ the dye acts as ^ conu^ing. 
20 * agent allowing tiie heart to be located* by imaging th^ dye flowing through it The; 
image of the heart, is then "frozen^ at a ppiht in tune when thb dye location (and 
; concentration), provides the best image. Often, this siiapshot image of tiie heart is 
triced oh a transparent overlay which is then sijperi^ After 
the dye concentration decreases, the overlay may be used to trackthe catheter locatioii 
25 • and movements relative to ^ heart, as indicated by the bverlayi' Tnere .are two 
disadvaiitages to this process; ' First,* the image of the heart, iriade visible with the dye, 
is.only temporary, requiring the overlay to intcktfie catheter movement Second, if the 
patient's position changes, the heart imaging must be repeated., 
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in addition tb locaiing a catheter relaiivs 10 a he^ for an angtogran^ 
niany other medical protedures where an inteivenuonal device is inserted into a 
patient, iainost, ifnot all. siidiproceduresjocarins and positioning the intervent^ 
device widixh a patient IS extrsniely important.. 
5 -Multiple views; or x-ray orientations, of. a patient are pften desnable to 

. visualize objects and locaiionsin three dimensions. Although there sreinany possible 
combinations of such views, sothe are .fevored by medical pasonnel for particular 
procedures. For cardiac imaging, theright antCT 

oblique (LAO) views are common. Figure 1 shows a scheniatic.sectional depiction 
id Of a prone patient lOahdx-raysources 12and 14. the RAO and lAO view art taken 
: from the patient's right and len, respecavely. of the anterior (frbrit) of the.heait. The 
■ obUque angle, shown as angles 16 and 18, isAs measure of how farofftfae vemcalan 
' x-tay soOTce is oriented. The muitipie \4sws are often taken with a single X-ray source 
that is repositioned between positions 12 and 14. An imaging system that readUy 
f 5 • .iM^serits RAO and LAO yievi's allows medical personnel to view images in a femiliar 
. fOTriat as this is wiiat is presently standard m the ait. Multiple preferred views for 
other procedures might similarly be pre-defined; 

There are pitoblems with present practices; First,, x-rays and fluoroscopy 
jiroduce radiatioti, and Ae effect of diis radiation over the long term may be harmful 
This is (Specially true for the: medical, personnel subjected to repeated radiation 
espostires over the course of an entire career. 

In. addition, the radio-opaque dyeinay haiih the patient. For example, an 
.angiogran is often perfoiroed on patients rnth serious heart problems. Injecting 
cheinicals Aat may Cause sensitiN-ity, or an jdlerac reaction, directly into the heart of. 
such k patient may cause a serious problem, and there is the possibility of kidney 
damage from the dye. . . 
• . Finally, x-ray techhiaties require complex x-ray equipment, and the costly 
overhead associated with such equipmem. VnUe this may iiot.be a priinary coticera 
in a well equipped . moderii hD.spita^ is of concern in less, developed or remote 
30 locarions. 

Ultrasound basedimagiag methods offer some potential advantages over x-ray 
• based imaging methods. Ultrasound methods do. not expose the patieflTortnedical 
personnel to radiation and may reduce or eliminate the need for costly x-ray 
equipment. Also, the ability of ultrasound methods to directly image soft tissue and 
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blood vesselSt .\yiihout the need for radio-opaque dye as a contrasting agent, eliminates 
the potential dye-related hanns to a patient 

It would be desirable to have an imagms . system t^ 
as a partial substitution for, x-ray and fluoroscopy methods of imaging in procedures 
5 such as angiography iand tissue biopsies* Preferably, such a system, would present 
medical persoimel wth mtdtiple vieu's . that are ^d^^ 

' Patent NoM,175,228/issued to Steenwyk and Ghi 
1 979, describes a catheter locadng device (**the *22S pitcnt'')-^ 

induction coil adjacent to the cathieter iip,.and a remote sensing device to monrtorthe 
10 ampUtude and phase ofsignals induced in the coil,as a means o . 
location: However, an electricai signal from an induction coil. is not well stiited for 
detection by ultrasound imaging eqiiipment. The '228 patent does teach one technique 
for locating a catheter, but it lacks the ability to directly map that location to a 3-D . 
ultrasonic image of tissues v^ithin the body. Instead, such a device deteimines a 
IS "- catheter location relative to a position on thc.skin siirfe 

bathe magnitude of the signal received ^t the skin surfkce, It would, be desirable to . . 
locate the catheter relative to internal body tissues^ not the skin. 

U^- Patent No. 5,5 15,853, issued to Smithi et aL on May .14, 1 996, describes 
a S-D uhrasdirnd traicking system based on tri^ Usi^ig a 

20 network of ^lieascfourpie^oelectric transducers n 

pOTOhal computer OPG) as a digital controller;\he '85^^ 

relative distances between the transdiicers. However, like the *228 . pattern discussed 
above, an apparatus, based bii the ^ 853 patent lacks, the ability to .map the locapon of 
a catheter to a 3-D ultrasonic image of tissue.within the body. Iristefad, the reference 
.25 frame for catheter location iiiformatioh is the location of other piezoelectric 
transducers. For example, a series of transducers contairied in a chest,h;arness aroimd 
a i>aticnt i&ay bie tised to triangulate the position 
to the chttt harness, by measuring the distances to 

The *853 pateiit does briefly mention thepossibility of "overlaying'* transducer location 
30 information on a video loop, created from x-rays or ultrasoiind^ to facilitate 
visualization of the catheter locaripn. However, there is no teaching of how this 
overlay process might be perfbrmed. In contrast, an eitibodimem . of the present 
invention is directed to describing the catheter location relative to tissue within the 
body in one or more 2-D views. . 
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■ U.S. Patent No.'s 5,817,022 and 5,868,673. ("the *022 and *673 patents'*) 

issued to Vesely on October 6, 1998, 3Xid on February 9, 1999, respecuvely, are both 
continuations-in-part from the application that became the '853 patent Both also 
claim 3-D ulu^ound tracking of interventional medical instruments by triangulating 
5 between a transducer attached to the medical instrument and a network of transducers 
either inside a patient^s body dr. on the body surface. It would be desirable to simplify 
the iapparatus so that a nehyork of trans 

The *b22 patent locates a medical instrument, relative to the network of 
trazisducei^. in 3-D^, iand then displays a 2-D iiltraisotmd 
10^ system; The 'bis patentdescribesa"iinagingmddalitysy&t^ 

3-b oi 4-D, image data, an "iniage registration system" to register the position the 
[medical] instruinent within the image data and a "user interface" to perform a desired 
functiQn,'SUch as selecting a particular view for display. However, there are no detailed, 
descriptions^ of these elements, or teachings which would permit one of ordinary skilj. 
is. . in the art to create a system eombining such elements- 

The * 673 patent describes a system for performing surgery while trackirig the 
location of a rnedical devicerelative to a network of transducers, on orwithin the body. 
The medical device location is theii mapped to the network of trarisceivers, but not 
directly to any portion of the body tissue. 
20 Ultrasound image data for an object^ such as a heart, . coiisists of the 

"echodensity" sampled at multiple locations within an object This echbdensity can 
be thought of as somewhat equivalent to the color of intensity of a visual iinage. The 
echodensity is a function of the sound velocity -vvithin.the tissue, as well as the tissue 
... density. Although a3-D image of echodensity, \viih 

25 . dimensions, may lack the clarity of the sharp edges in a solid model, the gradations 
cbhtain a significant anioint of information about the tissue, A solid rnddelirig 
approach.tcf presenting a 3rD ultrasound image typically uses a direshold value. "That . 
. is, all echodensity values are divided into two bins, those above and those below the 
threshold. Valiies above; the threshold are opaque arid those below are transpareriL 
30 This direshold technique adds clarity to the wide variations of the "murky'* echodensity 
. data, by distinguishmg solid featttfes within the data, at the expense of the detail stored 
in those inany values. 

. TomTec Imagiiig Systems, formerly of Boulder, CO, now TbmTec Imaging . 
Systems Gmbtt of Munich, Germany, produced a video tape describing a 3-D 
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ultrasound imaging system (*'ihe Tomjec. system")* TomTec Imaging systems, 
Clinical Three-Dimcnsionai Echocardiography, VHS/h^ 

(video tape). The TomT^c system: creates a 3-D cardiac image, with the use of a 
ultrasound transducer in the form of an endoscope, and produces an image of the 
5 surfaces of the heart, essentially a solid model. That is,-th0 surface of the hean is 
displayed without any iiiformaiion about the internal musde tissue. The. TomTec 
system does not appear to have been coinmefciilly successfial. . . ^ 

. - SURlNlARY dF THE IN^ENTHEON , 

iO ' ■ - • . • ■ ^ - . . • 

The ultrasound imaging systeni of the presem invention superirn 

views created froni volumetric ultrasound data and th^ 

•device, such as, but not limited to, a catheter. Tne position of catheter may be shown, 
in one or more views, reiadye to organs aiid tissue 

15 moved. The catheter positional data is updated cbntinubusly and. is superiinposed on 
tissue irnages. that may be updated less frequently^ resulting in real-time or hear real- 
time iihages of the catheter relative to the tissues. 

. The superimposed images allow medical personnel to perforin procediiressuch 
a^ angiograms vn(ti minimal exposure of patients to x-rays and contrasting dye. 

20 HbSveverfc the took and feel of die . familiar fluoroscopy-like imaging liiay be 
maintained. 

BRIEF DESCREPTION OF THE DIV^^^ 

Figure 1 is a scheihadc secdonai depicdoii of a padeht and x-ray-equipihent. 
Figure 2 is a diagram ofa 3-0 iinage space in 
25 of the present.invention.- 

Figure 3 is a scheihatic diagraiii of point location artiong reguliarly spaced 
image data samples in accordance with an embodiment 6f the jpres^nt invention: 

Figure 4 is a schematic diagram ofirregularly spaced image data samples in 
' accordance with an embodiment of the present invention. 
30 Figure 5 is a- schemadc diagram of a control panel in accordance with an 

embodiment of the present invention. • 

. Figure 6 is a flow chart of the imaging process in accordance with an 
embodiment of the present invention. 

■ .5- 
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DETAILED DESCRIPTION 

The present invention is directed at superimposing an- image of an. 
interventional medical device, such as a catheter, and a three-dimensional (3-D) 
ultrasound image of an orgaii or tissues, such as a heart, and creating one or more 

.5 projections from die superimposed image data The two sets, of image data may be 
updated at the same or different freouehcies. The image of the medical device may be 
• updated more frequently to show its current location, as the device is guided or 
repositioned, within a patient by medical personnel. The projections, made from the 
superimposed images pfefeftbly reflect the cur^ntlocation of the medical device, and 

10 depending on the .frequency of updating the image of the brgan or tissues, may also 
: show a real-time itfiage of such organ or tissues. 

The present inveriiibn may be used with a wide variety Of interventional 
medical devices, such as, biat not limited to, catheteirs* probes, caimulas, stylets^ aiad 
needles, and is not intended to be limited to Use widi any particular interventional 

15 medical device. For convenience, this disclosure will use theterm catheter, instead the 
more generic term interyentioh medical device, since the example description is related 
to use of die apparatus and me&od of ]iie invration i^ 
is not meant to imply any limitation on the scope of the invention. 

The present invention may be usedin connection with ultrasound iniag^s of any 

20 orgahi portioh of an organ, or - group , of prgaiiSv and the present invention is not 
intended to be limited to aiiy particular image subject matter, there are situations in. 
which it may be desirable to use the present invention in conjunction with x-ray 
methods. For example, if an ultrasound transducer is positioned so that only locations 
in the vicinity of the heart, can be. imaged, x-ray methods might be used to track a 

25 catheter from its insertion point near the groin until it comes into view with the 
transducer. ; 

The present invention may also be used with "4-D ultrasound images''. That 
is, repetitive sequences of 3-D. images in titne, such as, but not limited to, images of 
•a heart beating as it goes through one.prmpre cycles. The 4-b ultrasound images niay 
30 be. depicted and displayed as :repetiiiye sequences of .iniages, or cine loops, 
Aitematively, a.single point. in .the.. time sequence niay be selected, effectively 
. transforming the 4-D image into a S-D image. Preferably, the present invention allows 
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updates of the catheter location at a frequency independent orthe repetitive sequence ^ 
frequency and the frequency of individual image frames v, ithin the sequence. This 
- disclosure will use the term "dynamic display" when 4-D irnage data is distinguished 
from 3-D image data; Otherwise, the tenn "image data^' may apply to either 3-1) or 4- . 
5 D image data. 

The outpiitofthe present invention is one or more projectiotis created from the . 
superimposed ima^e data.. Although the projections are typically displayed oh bae or . 
more cathode ray mbe (CRT) monitors, the present invention may output the 
projections to aiiy type of graphic display de vic£(s), such as a thiri film transitor flat 

10 iscreen display, or to a graphic output file for storage or funher image procesising. The 
present invehiibri is nbt intended i6 be limited to use^viih any particulM" type of graphic 
display device or output file format Multiple projections, typically, tajcen from 
differing orientations with respect to the image data, are often beneficial in visualizing 
* Jocationsin a 3-D.space. Ah embodiment of the present invention uses two standard 

15 .views, rirfit anterior oblique (RAO) and left anterior oblique (LAO) as **defa;ult'* 
projection orientations;. 

As the trend towards faster computer processors continues, , eventually the 
present invention will' allow for real-timd or near teal-time updates of projections 
created .from 3-D image data! At present, the acqiiisition of. image data, and the 

2Q projection processing discussed below, takes too long for such real-time updates. 
Given the ciirrent constraint on computing power, the.present invention is preferably 
used in an environment where the patient is imrnobilized. Tnis penriits tise of a single 
set of image data may be used continuously whiie a catheter is moved relative to the 
pateiit and the image data. ' - ' ' [ - 

25 For purposes ofthe examples described beldwv references to a dy^ 

embodiment of the present inventioni presumes that the position of the organs and 
tissue return to the saine position and orientation within- a patient. at the end of each 
repetitive sequence, or cycle* That is, the cycle may consist of "local" movement but 
no "global'' movement relative to the immobilized pktiem is anticipated.; In contrast, 

30 movement of fee catheter is aiticipated, so the: location of the catheter is updated. 
frequently, preferably in real-time. The Updated catheter locatioii.is then superimposed 
on tiie relatively static organ of tissue imageLdatalhefbre the. projections are created. 
Cpiiecting 3-D ultrasound image data, while certainly not. a trivial procedure, 
. is known to those bf skill in the art,;and is not intended to be an element of the present , 
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invention. Ratheri the present invention uses such data, however collected^ ias input. . ^ 
.U.S; Patent No. 5, 1 8 1,5 14, issued to Solomon, et aL on January 26, 1 993, describes a 
transesophageal echo cardiography transducer device which may be used to collect 3-D 
ultrasonic image data ("the *il4 patent'*). This device, widi a rotatable ultrasound 
5 , itransducer array, may be inserted into a patient's esophagus and positioned adjacent ^ 
to the hekrt. A motor and drive mechanism, which remains external to the patient, is 
iised control the probe with the transdii^er. array and obtain 3-p image data. A 
transesophageal transducer, such as described in the *5l4 patent, might be usied to 
collect ciardiac image data. As the present invention is not limited to any psgticular. 
10 method or apparatus for collecting image data, it is intended thait as advsmces are made 
in ultrasound data collection, the present invention may utilize such improved image 
data as input \ " • 

image data will preferably take the fonn 

15 Or, in the case of image data for a dynamic display,, in the form of the 4-D array: 
. V . l{x,y,z,t) 

Where I represents the echodensity, or "intensity/'.of the ultrasound image at a point 
defined by the 3-D Cartesian coordinates x^y and Zi The particular choice of 

orientation for the a:, y and z axes does not fjfifect the present invennbn. Preferably, 

20 the axis orientation is used consistently. For a dynamic display, / refers to the instant 
in time representing when the echodensity data was sampled within: the repetitive 
sequence, Image data may be collected in different format5,.o 

then converted to the preferred format using transfoniiation formulae known to those 
of ordinary skill in the arL / ; 

25 In one eriibodijnentof the present invention the I values at each sampled point 

in space will . be' binary representations of the echodensity yaliies. The values may be 
in either integer or floating point format. Although integer values are typically better 
suited to efBcient tnathematical aJgorithmSj this efficiency rriay be effectively lost if 
. . numerical iriterpolatronmethodstising floating point values are selected for integration. 
30 The need for humeri caLintexpolation andintegradbr^^ 
. The / values, for moriochrpmatic embodiments of ih^ 

map to a spedficgray scale value that will be used to display the image. For example. 
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eiaht bit integer / values may represent 25 5 gray scaie values, from 0 to 255. Floating 
point / values may be grouped in ranges, each of which corresponds to a gray iscale 
resolution of the display. Tne range of gray scai^^esoiution values chosen will depend 
on both the characieristics of the display (where higher resolution produces better 
5 imases) and the processing speed of the image processing device (where higher 
resolutions require more calculations per pixel). Gray scales for displaying images are 
. known to those of ordinary skill in the an.. Tne present invention is not intended to be 
limited io any panicular gray scale resolution. 

in addition to Ae number of gray scale values, or gray scale resolution, the 
10 sp^al resollttion of the input image d^ also effects both the quality of the display and 
the number of calculations required. That is, image data sampled at many points in 3- 
D siiace will generally produee.a better image than the. same object sampled at fewer 
points: The trade ofFbeirig that the additional image sanitjle points may require more 
calculationis, which typically requires a more complex knd rapid handling sj'stem.or 
iS slows the image processing. 

Although techiuques for collecting the ultrasound irnage data ^ 
scope of the present invention, and image data is treated as input tp the present 
inventioh, an embodiment of the present, invention does allow a user to . call for an 
iqjdate of image data as nccdei For example, if the medical personnel detect that a 
20 patient has shifted^ changing the relaitive position oif the object being imaged and the 
cadieter, new J (x^ y, z) values may be collected In another embodOTcnt of the 
present invention, a patient movement tharis more than a pre-defined, threshold amount 
may trigg«-acall for updating the image data- As long as there is little pf no relative 
movement bervveen the object being imaged and the catheter* there is no need to updatd 
25. the image data. Preferably, the catheter is located iising the same apparanis use^ 
coUect the SrDtiltras.ound image dat^ (.r, z) - the format of the caA^^^ 
data will depend somewhat dh. the t>-pe of catheter used, as well as the associated 
means of locating the catheter. " . ■ 

. The catheter location may cdrrespohd tp the' direct echodensity data depicting 
30 . the image of the catheter, including' any image ennahcements from .active, catheter 
devices such as a vibraribirmethiaLnisnt U;S: iPatenrHo: 5-,343;S65 , issued toGaydinecr 
arid Vilkomcrsan on Septembers, 1 994; describes a catheter vibratiiig mechanism 
(•*the * 865 patent''). A vipratitig mechanism, such as described in thie *'865 patent, may 
enhance the visibility of a catheter, or other interventional medical device, that is 
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attached to it in an ultrasound imaging sysierh. The present invention is preferably 
used with such an active catheter. Potential benefits include faster imaging of the 
location, a clearer image, and siich an active catheterreduces ]the required sensitivity 
for the imaging system. The present invention is not, hoWever, intended to be limited 
5 to use with an active catheter, or any paniciilar type of active ciatheter system. The 
. image of the catheter location may also be enhanced, for better visibility^ through 
techniques such as brightening, increasing the contrast, or enlarging the catheter image 
before superimposing the catheter location data and the 3-D image data. With some 
ernbpdiments of the present invention, a false image, or icon^ may be used in place of . 
10 the catheter image data. Such techniques wiU be further diescribed below. It b very 
important, however, that the catheter location be matched closely to the coordinate 
syirtem of the image data. 

■ In -order td create projections of the S-D ijmage datai one or more projection 
; pltoes must be defined* Figure 2 depicts a 3-D space or yoium 
15 portion ofspace containing the imaged objects, su6h as drganswi^ : . 

cavity, and Could take virtually any 3-^D shape. Volume 22 uses (global), axes 24 io 
define a Cartesian cbordina!te system for briehtation and measuremienL A 2-D plane 
26. is selected relative, to volume 22, with (local) axes denoted as (il, v) . Typically, 

plane. 26 is defined by a unit vector /x, normal to (/f, v) . Unit vector. 72 can be 

20 defuiedas: 

and, because /i is a unit vector^ 

Where ,T,j> and f are xmit vectors along thex,> and saxes, respectively, and 
25 (/I/, 72 /zj)are the direction, cosines of n , Those of ordinary skill in the art will 

recognize that there are many mathematically equivalent ways of defining plane 26 and 
its normal vector n. \ 
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Imposing the condition; -4 

the directions of unit. sectoi3 u and v can be defined, using the constant i2, as; 

/ij, «i, i?) 


5 where; 


and; 


The construction of the projection on plane 26, can be calculated By pcrfonning 
id thcmtegratibri:- •■ •. , 

>(u, v):-Jl^y,z)dS : : ; • 

That is, the image data^ /(x^ jjr, r), is sumined along the line 

xohtbP(x/, v), whcrethevectbr 5 is an integrating line parallels 

by the.eqiiation: 
f5 x^.x^ -i-Sn 


The starting point, 5^, is given by:: 

^ Ri'^ uns^ yn$ ni^ uht^ vni ui^vi^^ 

20 and,- 

where /r is an integer andcj is the integration interval used, for the' nurnerical 
implementation. 


BNStXXlb: <WO_0lSa359AlJ > 


SUBSTITUTE SHEET (RULE 26) 


Wd 01/58359 PCT/USOl/00957 

can be thcHJ^ of as the thickness, or doDth, of mem v). 
■ In some embodiments of the present invention iS, and the starting point x;,^ set 
such that the projection is created from only a relatively thin slice of die image data 
I(x^ y ^ z). Preferably, the depth of the image data used in projection plane 6 is 

5 defined by the user along an axis parallel to h. 

The nimiericai implementation of the integration used to calculate 
v) may require interpolation. That is, the vector 5 and integration interval cr 

- niayresultinlocations Aatdoiwtdirectfy / C-x^ z). 

Referring now to Figure 3, which depicts a set of image data values 28, corresponding 
to to image sample locations witfiin I{x^ z), Valxies 28 are shown, for clarity, as . 

regularly spaced in three dimensions, although the same principals appl^ 
spaced image values. Location 30 is a point along x lised in the calculation of 
v), that does hot coCTespond to the location of a^^ 

skill in the art will recognize thait there are many techniques aivailable for assigriing a 
iS data value to loicatiori'30. . A. few of ftese techniqiaes are, in increasmg order of 
complexity, matching the value of the "nearest nei ghbor'*, avieraging nearby values and 
computing weighted averages* The present invention, however, is not intended to be 
limited to any particular inteipplation teidmique. 

tJ.S. Patent No. 5,226,1 13, issued. to "Glirie, et al. on July 6, 1993, describes 
20 using reverse ray casting to project volumetric images unto image plane rotated at 
kny arbinrary angle ("the * 1 13 patent"). The present invention takes 3-D volimietric 
image data, as one input, and projects 3-D image data onto; one or more image planes, 
which may be rotated relative to the axes of the 3-D image data. The present invention 
may use the projection techniques of die * 1 13 patent, or any another technique, and is. • 
25 not intended to be limited to any. particular method of projecting 3-P image data to 
create a 2-D vieW in an image plane. U^S- Patem 
. al. on February 2, 1 999, also described using reverse ray castmg to project volumetric 
images ("the V750 patent''). Like. th^ 

patent may be used by ah einbpdiment of the jjresent invention, to project 3rD image 
20 ■ data.-. . ■ 

The above projection a:eation procedure may be repeated for each desired 
projection display. Ah embodiment of the present invention allows the user to select 
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a fust projection, corrssponding to eiihsr an RAO or a LAO oricntatibii, at a user 
selected numfacr of degress off ihc verticil; wim a second projection (de&ult) 
orientation corresponding to the other. Preferably, defining me direction of the second 
projection orientation wouldrequire very littie input from the User, and instead would 
use translations firotn the first projection, orieaianqn to orient the second. 

PrKcnung projections to medica penonnel in a fam 
and LAO orientations, is merely one capability of the present invention^ Unlike x-ray 
• images taken from only one or two orientarions, me 3-D image data X (-X, y, z) used 
with the present invention may create vinually any view(s) a user might select. For 
example, a projection could be created of the tissue in front of a catheter, as viewed 
along the axis of thp catheter. 

The simplest fomi for the image data, is equally spaced imase samp les in each 
of the jc, and z directions. . Unequally spaeed,- or irregularly spaced, samples may 
aiiso be used with iflie present invention although.the calciilatipns may become more 
complex^ and.time consuming, irregularly spaced samples, inay also be pre-processed 
and converted to equally spaced image samples, by any one Of a variety of techmques, 
for use by aii embodiment of the present iiivemion.; An embodiment of the present 
mvcntioh may pre-process irregularly spaced /(:c, z) data values; Refeuing now 
to Figure 4, which depicts irregularly spaced set of irnage dat^ values 28, relative 
to a regular spacing location template 32. Daia values 28 may be used to interpolate, 
orCTcate, data sainples on tdmpiaie 32. Similar to the interpolation discussed above, 
the present invention is not intended to be limited t6 any parpciilar method 
interpola-ting or translating data values 28. Somc.projection algorithms may benefit 
from such a pre-processing of irregular image data into regularly spaced locations. 

A control panel for ah embodiment of the present invendpri is shown in Figure 
5. Joysticks 34 are used to choose iheorienmion of one orniore projectipris, and pairs 
of slide bars 3 6 choose the upper in lower limits of the" depth of the projections, S. . 
Brighmess 38 anci cdjitrast 40 controls allow the user to iadjiist the display .on one or 
more CRTs or other display de\-ices. Tnt present invention is not intended to be 
iunited to any panicular method'of allowing a User to choose a projectioh orientauon 
or control. the displays. Although direction cosines are usedin^ihis disclosure to 
describe projection piane,S; it is not intended for a user to be restricted to inputting - 
orientatidn information in that format. 
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in one embodiment of me preserii mvenuon, ine Jocauon oi we catneier . IS 

superimposed on the image data after the projecdon(s) h Generally, 
the creation of projections, P (n, v), from the image data file t{x, y, z) is a relatively 

' slow process^ Although it is understood that the trend towards faster and less" 
5 expensive computer processing will ease this constraint At the present time the 
projections are created less frequently than in real-time. This meaiis that as the catheter 
moves, first superimposing the catheter location arid J(a?, , z). and then creating one 

o^ more projections, P (u, v), may introduce a time-lag in displaying actiial. catheter 

location. Unlike the image da]ta file / z), which typically contains non-±ero 

10 data samples iat many thousands of locations corresponding.to the entire volimie of the 
imaged organs or tissues, the catheter lociation.^^ 

That is, it will contain riiiiqh fewer non-zero entries than J (x^y^ z). An einbodinient 
of the present inveiitioh exploits die fact that the catheter location data, is relatively 
^arse compared to J(a:, z), and superimposes flie catheter location direcdy on the 

15 projection(s). In this way, the changing posiuon of the catheter can be m 

the projection^ without the need to re-create the projection frorri each data point within 
/(;c, y^ z) . In most cases, the relatively sparse catheter location data can be directly 

... superimposed ori projections in miichlfess time than is requir^^ 

fipni z). . J / 

20 . Figure 6 shows a flow chart of an embodiment of the present invention where 

• . ' :the intervention medical device location is superimposed 42 after the projectioh(s) are 
created .44, ahd then the projection with the device location is displayed 46. The 
* process of locating the device position 48 is repeated frequently, preferably at a 
frequency close to the display monitor video refi^^ 
:25. . . . Re-starting the process, by obtaining Updated image data 50, may be initiated 

by the user in sitiiations such as where there is patient movement. With sufficiexit 
computer processing power, obtaining updated image data 50 may be: repeated at 
predetermined frequency without liser a^^ 

The location of the cadieter is preferably detected by the same ultrasonic 
20 : imaging^eq^iipment that is used to create the. image data, J(x, z)^ using the same 

global Cartesian coordinate system. By. using.an active catheter system, such as the 
\dbrating mechanisni of the 
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•865 patent, the "catheter image' may oe mors ezsijy aetectea, ana Qistmguisnea irom, 
the relatively passive isufrounding environment. An embodiment of the presdjt 
invention may also tise one or more filters to further distinguish the active catheter 
image from backgroimd: Such filtering techniques are generally knpwn to those or . 
5 prdinaiy skill in the ait. 

The catheter iifiage may take one of two preferred formats. First, it inay match 
the format of the input file I(x, y, z)Nviih only a relatively limited number of non- 
zero entries. : For the catheter iniage this file will be caU 

ifamati L{x, z\ vvill contain the echodensiiy values of the catheter, corresponding 
io to tiie actual 3-D image of 4is ca&eter, as seen by the ultjrasouhd transducer. As 

discussed above, the 7. c) values may be the post-filter echodensity data. 

Alternatively, the catheter imige may correspbiid to one or tnore $et5.pf 

Cartesian coorainates, rejjresenting the coordinates of one or more locaMdns on Ae . 

catbetef in 3-D space. Thatis<. the acoid uJtrasoufad image of the iqadieter is not used, 
IS ody the coordiriates of one or more points that may be med la defcimin 

location, and perhaps its orientation- 

Usiiig tvvo points on a catheter to locate and orient the catheter in space is 

khown in the art, as iUiistraied by Figure 5 in the '853 patent. All OT^ 

present invention includes assignable anribute data field for each coordinate, along 
20 wath thecoordiiiates of Ae oneormoreloj»tiDnsbnthecatheterl Such a (Jataiield 

inay store an inteiisity value corrcspondmg to an amoimt of ultrasonic ener^ detected,. 

for iise in creatirig false color displays of Ae catheter image. 

Regsffdle^ of the format of the catheter locatiori data, the p^^ 

petftwrns a transformation of that location data into the coordinates of the projection(s). 
25 . For citheter image files in the L{x, y, z) format, this process: may resOTble 
: (mathematically) the creation of P(tt, v) from /(jc, y, s) . only with a much sparser 

ihputfile..Theprojectedcatheterima2e(s) A(tt,:v)cani^^^ . 

on the linage data P(tt, v). This procedure is repeated, or performed as multiple 

parallel tasks, for each, projectibn orientation. 
30 For catheter locaiioirdatastored as.cpordmate(s)i or fcoprdinate($) and attribute 

.datai only the individual coordinates need to be transformed into the coordinate 
systems of the projections. That is, there is no need to perform numerical integration 
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on the larger number of data locations making up the catheter image. Such a 
transforination of one or more points to the coordinate, system of the projections itiay ^ 
be performed relatively quickly, in either software or hardware. The output of the 
transformation system may be two (planar) .coordinates, for each. point location in a 
5 given projection, and the values of the associated attribute data. Mapping a locaribn 
symbol to a CRT display at the location of these two coordinates is straight forward, 
and known to those ofordinary skill in the arL The atrribiite data may also be used in 
conjiinction with the location symbol. For exaihpl e, "cross-hairs", or an arrow symbol, 
could be mapped onto ]the CRT display of each projection at the current location of the 
10 . catheter in that proj ection, with the color of the symbol correlated to the magnitude of 
vibration detected in a vibrating catheter. ' . 

All embodiment of the present inventioti converts catheter image data in the 
form of a I^ix, z) into one'pr more coordinates representing the catheter, location 

in order to speed the data processing^ The full ulirasbimd image of the catheter is 
f 5 sicrificbd in siich an enabodiment, for the ease in processing data for drily a few point 
locatidns- However, widi one or two point locations, and the prior knowledge of the 
catheter geometry, there is ho need to directly map the full catheter image to the 
display; all of the infoiroatibn can be carried by, or conveyed by, the symbols mapped 
directly to the monitors, 
20 Unlike the projection data created from the image data/(A:, ^, r), the 

superimposed catheter image does not require that the image interisities (gray scales) 
be accurately transformed unto the displays, only that the catheter locations be 
accurately transformed. It may be advantageous to modify the intensity of the catheter 
image, relative to the other image data, for display clarity. One embodiment of the 

25 . present invention iises the catheter Idcatioh as the center of projection depth when for 
of eating the image for display. 

- Another embodiment of the; present inventiori may use color image data files, 
where /(x, >, z) is replaced by separate color spa<:eimage files, such aslt^r, j^,.r), 
litiXyyj z) and Is(x^ z) corresponding to an iinage in RGB color ispace. Other 

30 color space formats may also be used, and the present invention is hot intended to be 
limited to any particular color space representation. Given this discloisure, those of 
ordinary skill in the art vVill recognize how the present invention may be adapted to 
work with such color images^ 
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Although various embodimehis axe specificaily illusjraied and described herein, 
it will be appreciated that modifications and variations of the present invention are 
covered by the above teachings and are with the purview of the appended claims, 
without departing ifroni spirit and intended scope of ihc present invention. Moreover, 
5 the present invention may be implemented in software, firmvyare as \veU as in 
hardware^, and in ia broad: range bf computer architectures and is intended to be a 
general approach that includes a broad range of specific implenaentatiphs. 
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CLAIMS 

What is ciairried is: 

1 h An apparatus for .<Usplaying ultrasound voliunetric. object data and an 

2 interventional device, comprising:* 

3 a volumetric data file for storing a plurality of object intensity values, each of 

4 said intensity values mapped to a set of 3*D coordinates; 

5 . a first projection plane for.displaying a first projection of said object intensity 

6 values, whereby said first projection plane is defined by a first peTpendicular vector; 

7 a first iniage projector for creating a first projection image oh said first 

8 projection plane from said object intensity values projected along said first 
. 9 peipehdicular vector; • 

10 an. interventional device locator for locating the . inter\'entib 

11 device location mapped to said set of 3-D coordinates; and 

12 a first superimpdseir for superiinpbsing said device location and said first 

13 projection image to create a first display image. 

1 2, An apparatus in accordance with claim 1, furth 

2 a second projection plane for displacing a second projection of said object 

3 intexlsity : values whereby said second projection plane is . defined by a second 
■ 4'- perpendicular vector; 

5 a secondiniageprbjector for creating asecondprojectionimage on said second 

6 projection plane from said object intensity values projected along s.aid second 

7 perpendicular vecton and 

6 a second superimposer for superimposing said device location and said second : 

9 . projectioh iinige to create a second display iniage, 

1 3. An apparatus in accordance with claim 2, wherein: . 

2. a preferred trahsfoimation defines an orientation relationship between said first 

3 ' perpendicular Vector and said second perpendicular yect^^^^ 
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1 4. . An apparatus in accordance with claim 1, wherein: 

2 said first display image is a view used in common practice by medical 

3 personnel performing a procedure. 

1 5. An apparatus in accordance with claim 2, wherein: 

2 said second display image is a view used in common practice by medical 

3 personnel performing a procedure, 

1 . 6. . Aii apparatus in accordance with claim 4, wherein: 

2 , said first display image is a right anterior oblique (RAO) view of the object. 

. 1 7. An apparatus in accordance wirii claiin 5,. wherein: 

2 siid second display image is a left anterior oblique (LAO) view of the objecc 

1 . 8- . An appauratiis ih.accoi'dance wi]th claim 4, where^^^^ 

2 said first. display image is a LAO vie\v of liie object. 

1 9. Aii apparatx^s in accordance with claim 5, wherein: 

2 . said second display image is a RAO view of the object, 

1 lb. Ail apparatus in accordance with claim 1,. further comprising: 

2 a first image range, value for specifying a range of said object intensity values 
.3 in said first display image. 

1 11. An apparatus in accordance with claim 2, further cpmprising: 

2 a second image range value for . specifying a raiige of said object intensify 

3 values in said secpnd display image* 

1 12- • An apparatus in accordance with claim 10, wherein: 

2 said first image range is centered on said interyehtidnal device location as 

3 measured along said fint perpendicular vector. 
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1 13, All appaianis in accordance with claim 1 1 , wherein: 

2 said second image range is centered on said inierventibnal device location as 

3 measured along said second perpendicular vector. 

1 14.. An apparatus in accordance uitH claiih I, further comprisb^^ 
■ 2 a first display image oriclntation control for defining said first perpendicular 

3 vector firom a tiserihpuL. 

1 15. An apparatus in accordance with claim 1, wherein: 

.2 said STLiperimposer updates said device location cdntinuo^^ 

. -f 16. An apparatus in accordaiice with claun 15, wherein: 

2 said first image projector updates said object intensity values responsive to a 
.,3 user signal. 

f 17. Ah apparatus in accordance with, claim 15, whe^^ * 

2 said first image projector updates.said object intensity values in .real-time. 

i 18. An apparatus in accordance. with claim 1, wherein: 
• 2 the iilterventional device actively transmits a signal and said interventional 

^ device locator image generator is responsive to said signal. 

1 19. Ah apparatus in accordance \viihclairh l, fW 

2 an. output file generator for storing said first display image. 

' i . 20. . An apparatus in accordance with claim 2, further comprising: 

2 a thirdprojectionplane for displayiiig a third projection ofsaid object in tensity 

3 values whereby said third projection planeis defined by a third perpehdiciilar vector 

4 a third image projector for creating a third projection irhage on said third. 

5 projection plaiie from said object intensity values, projected ^lorig said, third 

6 perpendicular, vector; and 

7 a third superirnposer. for superirhposmg said device location and said third 

8 projection image to create a third display image, 

'20- 
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1 21- An apparatus lor cispiaymg uiirasouna voiumetnc bojeci w2Ui ana an. 

2 : . .interventional ^ 

3 device, comprising: . 

4 a volumetric data rile for storing a plurality of object intensity values, each of 

5 said iiitensity values mapped to a set of 4-D coordinates; . . 

6 a first proj ectipn planefor displaying a first projection of said obj ect intensity. . 

7 values, whereby said first projection plane is defined by a first perpendicular vector, 
' g 2i first image projector for creating a first projection image On said first 

9 projection plane from said object intensity values . projected along said first 

10 perpendicular vector; 

11 an interventional device locator for locating the interventional device." at a 

12 device location mapped to. said set of 4-D coordinates; and 

. ^3 a iirst superinlposer .for superimposing said device location and said first 

14 projection image to create a first display image. 

1 22. An apparatus in accordance x^ith claim 21, wherein: 

2 said first display image is a cine loop. 

1 25 . An apparatus in accordance yvith claim 2 1 , wherein: 

2 said first display image corresponds to a single point in a rep^^ 

1 24. A method of displaying ultrasound voluihetric obj ect data and an inter\-entional 

2 ♦ device, comprising: 

3 selecting, a plurality of object intensity values mapped to a -set of 3-D 

4 coordinates; . . 

5 sclectmg a first projection plane, for displaying a projection of said object 

6 intensity values, whereby said first projection plane i^ defined by a firs: perpendicular 

7 . viector; 

8 projecting said plurality bfbbject intensity values 

9 vectdr prito said first ipfojectiohpl^ 

10 finding an .interventional devi^qe location rdati^^ • 

. 11 and . 

12 ' superimposing said device location andsaid fir^t prbjection plane onto a first 

'21* 
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1 25. A method in accordance wiih claim 24, further comprising: 

2 selecting a second projecnon plane for displaying a second projection of said 

3 . object intensity values projecting a second display, whereby said second projection 

4 plane is defined by a second peipendicular vector; and 

.5 projecting said plurality pf object, intensity values along isaid second 

5 perpendicular vector onto said second projection plane; and 

.7 superimposing said device location and said second projection plane onto a 

8 second display image, 

.1 ' 26.. A method in accordance with claim 23 J, wherein: 

2 selecting a second projection plane is performed using said first perpendicular 

3 vector arid a preferred transformation. 

1. 27. A method in accoridance with claim 24, wherein: 

2. sclecfing said first projection plane conforms to a common practice by naedi^^^ 
3 personnel performing a procedure. 

1 28. A niethod in accordance wiSi claim 25, %\^erein: 

2 selecting said second projection plane confonns to a^^ 

3 . medical persoiinel performing ia prodedure, 

1 29, .Amethddinaccbrdi^cewithclaiih27,^erei^^ ' 

2 . said first projection plane is a right anterior oblique (RAO) view. 

1 . 30. A iriethod in accordance with claim 28,* wherein: ' 

2 said second projecnon plane. is a left. anterior oblique (LAO) vie 

1 31. A method in accordance with claim 27, wherein: 

2 said second projection plaiie is a LAO view. 
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1 32, A method in accordance wiih claim 28, wherein: 

2 said second jprojecdon plane is a RAO view. 

1 33. A method ia acccttdauace.mth. claim 24, further comprismg: 

2 selectingafimimage.range Value for limiting said pliiralityo^ 

3 values in said first display image. 

1 34. A meAod in accordance with claim 25^ further 

2 . selecting a second image range value for limiting said plurality of object 

3 intensity values in said second display image* 

1 . 35. A method in accordance ivith claim 33, further comprising: 
. 2 centering said first display image on said device location, as measured, along 

3 said first perperidicular vector, ' . . . . ^ 

1 36.: A method in accordance with claim 34, farther comprising: 

2 centering said second di^lay image on said device location, as measured.along 

3 said second perpendicular vector. 

1 37. A method in accordahce with claim 24, further comprising: 
2 , • . repeating saiid finding of said device locauori continuously. 

1 38. . A method in accordance with claim 24, further comprising 

2 selecting a revised plurality of object intensity values in response'to a liser 

3 signal. ■■ :' \ . 

1 39. A method in accordance \yith claim 24, further comprising: 

2 selecting a revised plurality of object, intensity values continuously. 

1. .40. A method in accordance wiich claim 24, fiinher comprising: 

2 seibcting a revised plurality of object intensity values in response to amotion 

3 greater than a threshold value. 
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1 . 4L A method in accordance with claim 24, funhei comprising: 

2 selecting a third projeciipn plane fordisplayinga third projection of said object 

3 intensity values projecting a third display, whereby said third projection plane is 

4 defined by a third'perpendicular vector; and 

5 projecting said plurality of object intensity values alongsaid third peipendicular 

6 vector onto said third projection plane; and 

7 superimposing said device location and said third projection plane onto a third 

8 display image. 

1 42. A method of displaying ultrasound volumetric object data and an interventional 

2 device, comprising: 

3 selecting a plurality of object intensity values mapped to a set of 4-D 

4 coordinates; 

5 selecting a first projection plane for displaying a projection of saiid object 

6 intensity values, whereby said first projection plane is defined by a first perpendicular 
;7 vector; 

8 projecting said plurality of object intensity values along said first perpendicular 

9 vector onto said first projection plane; 

10 finding an interventional device location relative to said set of 4-D coordinates; 

11 and 

12 superirhposing said device iocaiion and said first projection plane onto a first 

13 display image. 

1 43, A method in accordance with claim 42, wherein: 

2 superimposiiig said device location is performed in a cine loop. 

1 44. A method in accordance with claini 42, wherein: 

2 superimposing said device location is performed at a sihgle.point in a repetitive 

3 sequeiice. 
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